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[57] ABSTRACT 

A triangulation-based method and system for high speed 3D 
and gray scale imaging and associated pre-processing of 
digitized information allows for estimation or filtering of 
height and gray scale values based upon the confidence level 
of the infonmation obtained from a pair of sensors and also 
based upon knowledge of the object structure and its reflec- 
tance characteristics. A modulated laser beam is scanned 
across the object to create a plurality of spots which are 
viewed by a pair of well-matched receivers. Each receiver 
includes a light collection and delivery system, a position 
sensitive detector, and an associated ratiometric signal 
processor, or similar means for extraction of height and 
intensity information or data by triangulation. An optional 
automatic light control sub-system provides greatly 
extended dynamic range with control inputs derived from an 
amplifying detector included in each receiver to maximize 
occurrences of valid data points. Prior to image processing, 
each point in the digitized 3D and gray scale data is 
transformed based upon the digital values of the gray scale 
and height values. The multi-channel pointwise transforma- 
tion produces an improved estimate of intensity and height 
at a point of interest, or is used to filter the data for 
improvements in the 3D and gray scale contrast prior to 
image processing and measurement analysis. Either linear or 
non-Unear transformations are utilized, based on the match 
between 3D and gray values, and other conditions which 
determine the confidence level of the information. 

20 Claims, 6 Drawing Sheets 
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TRIANGULATION-BASED 3-D IMAGING A three dimensional imager which can meet accuracy 

AND PROCESSING METHOD AND SYSTEM requirements and speed requirements has a recognized need 

in at least the following areas; 

This is a continuation of application Ser. No. 08/658,579 Inspection of circuit boards and components: solder paste 

filed on Jul. 29, 1996 now U.S. Pat. No. 5,654,800. 5 and components, lead and solder ball grid array coplanarity, 

package dimensions and lead to body standoff; 

TECHNICAL FIELD Microelectronics assemblies: multi-chip-raodules, high 

This invention relates to machine vision methods and density miniature bump grid arrays, TAB inner lead bond, 

systems and, in particular, to triangulation-based 3D imag- wire bond; 

ing and processing methods and systems. Micromachined parts: miniature machines and mechani- 
cal assemblies; 

BACKGROUND ART D^ta storage industry: disk platter flatness, disk drive 

3D imaging techniques have been demonstrated ranging suspensions, defect detection; 

from con-focal microscopes to time-of-flight laser General surface defect detection: ceramics, glass, metals, 

rangefinders, covering applications from microscopic inte- paper; and 

grated circuit measurements to large field-of-view applica- Shape analysis and matching: forensic science including 

tions like mobile robot navigation. Performance specifica- toolmarks, firearm ID, pattern matching, 3D texture analy- 

tions for these sensors vary widely, with a figure of merit sis. 

often determined based upon specific requirements. Of all known methods for 3D imaging, triangulation 
Many applications impose simultaneous requirements of provides the most practical method for a high speed- 
rapid inspection rates for 100% inspection of the entire accuracy product. Triangulation is the most effective method 
object volume, yet accuracy and precision measured in for applications where a relatively short working distance is 
microns or perhaps finer. Such requirements present chal- acceptable and where off-axis viewing of three dimensional 
lenges for sensor design, and necessary consideration of ^ shape is tolerable (i.e. no narrow, deep holes), 
fundamental imaging limits. These limits require not only FIG. 1 shows a sketch of a typical triangulation based 
examination of the technical specifications of scanning and sensor. An incident laser beam 10 strikes an object 12 at a 
detection devices used for image formation, but close first or a second height 14 and 16, respectively. A position 
examination of the characteristics (reflection, transmission, sensitive detector or array sensor 18 is provided as well as 
size relative to sensor resolution) of the objects being a receiver optical system 20 for magnification. The instan- 
exarained. In a 3D system, the reproducibility of a measure- taneous field of view of the sensor 18 is indicated at 22. 
ment is often limited by sensor parameters (signal to noise. Height and gray scale information are provided on lines 24 
resolution, etc.) but the absolute accuracy is limited by the from a signal processor 26 to a computer 28 for image 
object structure and the influence of optical background processing and analysis. 

(both spatial and temporal noise. 35 Although the triangulation method provides many 

With increasing trends toward miniaturization, manufac- advantages, it is not immune to certain types of errors, the 

turers of microelectronics assemblies and miniature parts most obvious of which are due to shadowing or occlusion as 

often have requirements for very fine measurement or defect shown in FIG, 2. These problems are compensated for by the 

detection capability for automatic three-dimensional optical use of two or more sensors 30 and 31. An incident laser 

inspection equipment. 3D sensors which can potentially 40 beam 32 strikes a step object 34 which has an occluded 

satisfy the measurement requirements include confocal region 36 wherein sensor rays from the surface of the object 

microscopes, dynamic focus sensors, triangulation-based are blocked from the sensor 30. However, this region 36 is 

probes, and various interferometric devices. Cost effective visible to the sensor 31 as indicated at line 38, The IFOV of 

solutions for process monitoring where specific regions of both sensors 30 and 31 is indicated at 39. 

interest are monitored now exist, but are often limited in 45 More detailed examination of potential defects shows 

measurement speed. other important errors arise from acceptance of scattered 

For example, the requirements for 100% inspection of a light and various sources of optical/electronic background 

6"x6" wafer (or other object) with a 2.5 /mix2.5 /mi (which noise within the instantaneous field of view (IFOV). In 

can be coarse for inspection of such devices) samples taken triangulation systems, unlike dynamic focus sensors, the 

over the entire surface are extreme. A typical dynamic focus 50 IFOV along the triangulation sensing axis cannot be reduced 

probe acquires about 1000 data points per second. The to a single point, and the system designer must be prepared 

resulting inspection time for a 6"x6" object is 1.3 months. to deal with this limitation. 

Point triangulation sensors operate at about 10000 points per It is important to recognize that simple occlusion is a 

second and reduce the inspection time to a few days. 3D relatively easy problem to solve with just a pair of detectors, 

sensors which approach video rates, as described in U.S. Pat. 55 as shown in FIG. 2. Often a sum or average of recorded 

No. 5,024,529 reduce the inspection time to minutes which height values is used for an approximation, and if the 

is acceptable for many facilities. shadow is perfect (no stray light leakage), the proper value 

Also, other applications for in-line 100% inspection may will be recorded. Also, error contributions created by non- 

require a few seconds for 100% inspection of a 6"x6" part symmetric spot distributions resulting from surface 

with coarser resolution (say 25 ^mx25 /im. In many 60 roughness/marks in non-occluded or quasi-flat areas are 

applications, a tradeoff between speed and accuracy is not significantly reduced by averaging the asymmetric spot 

acceptable which restricts many present measurement tools distribution seen along the position sensing axis. This is a 

to off-line inspection which gready reduces the value added. consequence of imaging from opposite viewpoints where if 

Hence, accuracy and high speed, which are mutually con- the peak of the asymmetric spot distribution appears high to 

flicting parameters, are both required to maximize the return 65 detector #1, it wiU appear low to detector #2. The average (or 

on investment of inspection equipment. Clearly, very rapid midpoint for two detectors) is a good estimate of the spot 

data rates must be achieved for 100% inspection, center, and the error cancels or is greatly reduced. 
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However, examination of complex surfaces indicates that exceed the dynamic range of the system, even if high speed 

sensor processing techniques which only add or average automatic light control (ALC) is used. If ALC is used to 

multiple channels do not eliminate or even identify some of bring the sensor 68 into range (say, by reducing signal by 

the most severe errors. In FIG. 3, an incident laser beam 40 one to two decades), sensor 70 may receive low light return 
penetrates translucent material of an object 42 and propa- s resulting in a poor signal to noise ratio and an inaccurate 

gates outward becoming visible as background light to a first estimate of the height. Although the sensor is brought into 

sensor 44 wherein the object 42 blocks light from a scat- the intensity range, the local dynamic range is so large that 

tering center invisible to the first sensor 44. At a region 45 optical crosstalk (smear and streak noise) may result in 

of the object 42, scattered light is visible to both detectors. corruption of the height values in the region near the 
The light is scattered from internal scattering centers 43. The 10 saturation point because signal may vary by about four 

average value no longer is a proper height measurement but decades in a local region. 

is erroneously recorded as a very high peak when, in fact, the Furthermore, if the ALC range is extreme, measurement 

region is much lower. True siirface position is indicated at 46 speed may be compromised. If the range of ALC is insuf- 

and a truncated (shadowed) spot distribution is indicated at ficient and compression of the strong signal does not pro- 
48. At a second sensor SO, the spot distribution is shown at 15 duce valid data for the sensor 68, then the sensor 70 will also 

52. At region 47, scattered light is only visible to the second be out of range. If direct laser diode modulation (only one 

sensor 50. A ray from the scattering center seen by the sensor decade range) is incorporated, this situation can easily occur. 

50 is indicated at 49. Hence, the strongest signal may not always be best. 

This severe error illustrated in FIG. 3 limits proper Scan #3. A third scan down the side of the object shows 
discrimination of the object and background on the basis of ^0 ^ ^^^^^ difficult problem. The specular surface illuminates 

height, and even capability for "presence/absence" detec- what may be a glossy background which, because of the 

tion. ' angle of incidence and reflection, produces an intensity 

Similarly, FIG. 4 shows a case often found on circuit reading at the sensor 68 which is, perhaps, orders of mag- 
boards where a series of pads 54 are covered with solder nitude stronger than the signal from the point of interest. In 
paste or, on a finer scale, where leads are connected to the ^ this case, both the height and intensity readings from the 
bonding pads in the inner lead bond region of a tab inte- sensors 68 and 70 are much different and indicate a severe 
grated circuit. FIG. 4 shows first and second laser light scans error condition and low confidence in the data. 

51 and 53 and their associated first and second detectors 56 These illustrations show that for scanning of a single 
and 58, respectively. A line 60 indicates current detector solder ball, each sensor is required to operate throughout the 
position and spots on the detectors 56 and 58 represent entire dynamic range, and will encounter spurious data at 
actual detector position. many points. For measurement of grids, the luxury of 

The pads 54 are in close proximity, and the slanted edges re-orienting the sensor through rotation of articulation can- 
lead to multiple reflections from adjacent pads or secondary not be afforded if 100% inspection is to be accomplished at 
background illumination as indicated at line 62. Lines 64 assembly line rates. Furthermore, for such a geometry, no 
indicate useful distance reflection from the point of interest. favorable position for the sensor may exist which eliminates 
The sensor sees a secondary reflection from adjacent pad the error. Nevertheless, key information for the measure- 
resulting in an erroneous high value, but the noise is ment task is stiU available, provided erroneous data points 
occluded by the sensor 58, so the error cannot cancel with can be recognized or filtered. Many other examples can be 
simple averaging. Hence, there is an artifact at the edge found where simple averaging of height values (suggested in 
manifested as an incorrect high Z value. prior art) produces large measurement errors, particularly 

Another important example is shown in FIG. 5. The object for complex shapes with smooth surface finish, 

has a shiny surface finish and a curved geometric shape, the Errors associated with these phenomena have been quan- 

configuration which is most challenging for any 3D imaging tified. A typical problematic case arises when inspecting 
system. A specific example is a single solder ball 66 which 45 solder paste and traces on printed circuit boards. The effect 

is an element of a ball grid array (BGA) consisting of of occlusion and reflection artifacts is best illustrated by 

hundreds or even thousands of solder locations per square comparing results for long, thin objects which are measured, 

inch. Dimensions of each solder ball can vary from about ten then rotated by 90 degrees, and measured again. Test results 

microns to about 0.5 mm. With a high density of small indicated variation in readings firom about 30% to 50% 

diameter features, inspection wiU only prove useful if the depending upon the reflectance variation between the solder 

imaging and image analysis system are able to provide paste, background, and pads. 

measurements indicating deviation of coplanarity (height) The error is noticeable when the occluded region as 

and ball diameter measurements at a very rapid rate of illustrated in FIG. 4 is along the length of the pad, corre- 

speed, perhaps several thotisand per second. Examination of spending to the position sensitive dimension. The artifacts 
three representative scans across the object illustrate chal- 55 not only arc manifested in the computation, but are so large 

lenges in imaging and preprocessing. as to be visible to the eye. Such variation is unacceptable for 

Scan #1 (not shown in FIG. 5). When the scanning laser in-line process control, and because of the visibility, barely 

beam is near the peak of the solder ball, the gray scale and acceptable for "presence/absence" (coarse inspection), 

height values will be well matched and well within the Many triangulation based probes are available and can be 
dynamic range of the sensor, and the average height value is go broadly classified as foUows: point sensors, line of light 

a good estimate and cancels some asymmetry in the spot systems, and laser scanners producing data rates of about 

distribution. The surface orientation is favorable for both lOK points per second, up to lOOK points per second, and up 

sensors 68 and 70. to lOM points per second respectively. Regardless of the 

Scan #2. However, for a second scan for an incident laser particular embodiment, it is weU known among those skilled 
beam 72 where the surface normal is such that sensor 68 is 65 in triangulation based imaging that the use of various 

receiving direct specular reflection, a very high gray scale combinations of multiple transmitter and receiver devices 

value will be recorded at the sensor 68. This signal can easily can be used to improve performance through reduction of 
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shadowing and occlusion related defects as illustrated in direction orthogonal to the position sensing axis is blocked 

FIG. 4. Aside from the disadvantage of slow measurement resulting in an IFOV which is a narrow strip, similar to the 

speed, point sensors have been developed with 4 detector Penney and Stem disclosures. The position sensing axis is 

systems and 8 detector systems arranged symmetrically exposed and with the use of only a single detection device 

about the transmitted point of light. In 4 detector systems, $ the system is limited by phenomena illustrated in FIGS. 4 

the receivers are positioned at the sides square, and in a ring ^nd 5. The most severe errors arc not compensated. Light 

detector arrangement for 8 detector systems. There is obvi- ^hi^jji is blocked often has lower intensity than *^highlighu" 

ous benefit to this approach, but upon consideration of found along PSD axis 

inspection speed requirements, the method is not acceptable. ' 5,048.965, describes a 3D 

m an array camera system^e errors r^tmg from j^,^ ^.t^^,, using a CCD linear array and matched pairs 

reflection phenomena shown m FIGS. 3-5 can sometimes be r ,* . f c i ■ t-u 

reduced by counting and suppressing multiple spots found f °f avoidance. The method has po ential 

along the position sensing i^s, but only at the penalty of ^^.^^^g^^^P/f.^ acquisition (approachmg video rates) and, 

severely reduced measurement speed. Furthermore, the cor- with a 4096 hnear array, smiultaneous wide field coverage 

rect height value cannot be discerned simply by looking at With this method, two images, each with two matched light 

the intensity of each spot, sometimes the erroneous value sources, are required for occlusion avoidance. Instead of 

may have a larger intensity, as shown in FIG. 5. rising a position sensitive detector (PSD) for each receiver, 

Une of light systems offer substantial measurement speed equivalent function is provided by a pair of encoded light 

advantages over mechanically scanned point detectors, but patterns (illummation gradients of opposite sign) which arc 

are Umited by the camera readout rate to the 30K point per projected onto the surface and imaged onto the array, c^e 

second range. Multiple lines of light, high speed framing illumination pattern for each smgle scan of the object. The 

cameras, or smaller arrays which can be read at faster frame height measurement obtained from a single lUumination 

rates offer some advantages. Nevertheless, accuracy wiU direction is computed with ratiometnc processing (similar to 

often be unacceptable in measurement appUcations (as PSD based systems) applied to the two encoded patterns 

opposed to height presence/absence) unless scene dependent (which are switched between passes). For occlusion 

errors are reduced. In laser line of light or line scan systems, avoidance, a prescribed relationship between the intensity 

the symmetry which allows for simple averaging of non- values is used to compute depth as desired from the lUumi- 

occluded detector pairs is absent, restricting the number of nation spatial profiles. Exammation of the resulting equaUon 

detectors to 2 devices, unless successive scans are utilized shows the relationship is equivalent to averagmg the cen- 

with rotation of the imaging head or part, or additional troid estimates obtained from the two illummation direc- 

cameras and scan lines added for orthogonal viewing. tions. 

The Clerget, U.S. Pat. No. 4,534,650, describes a 3D The Chen et al., U.S. Pat. No. 5,118,192, shows a laser 
measurement device utilizing at least two distinct sensors based 3D sensor system with internal mechanical means of 
(photodiode bare). Part of the purpose of the part is to correct rotating sensors to achieve favorable viewing conditions of 
output from "faulty photosensitive elements". As can be irregular objects. The system utilizes multiple laser trans- 
seen from the above discussion of reflections and occlusion, mitters and detectors. The arrangement is optimized for 
that is the least of the problems which can occur. viewing of objects like solder joints where occlusion and 

The Case ct al., U.S. Pat. Nos. 4,891,772 and 4,733,969, reflection artifacts are similar as illustrated in FIG. 5. The 

describe the use of a "plurality" of sensors for line and point oblique illumination and viewing arrangement, together 

measurements. Features include exposure control based with a means of pivoting the sensor, aUow a substantia 

upon spot size measurement so as to aUow for use of nxn faction of the sloped sides of the solder jomt to be viewed 

array or sensors and "low level" algorithms for estimation of ^ such a manner that a high signal to noise— low spurious 

the spot center from the intensity distribution obtained from reflection condition— is achieved for typical surface onen- 

readout of each CCD array. In the ^969 patent it was stated tation and conditions. However, several conditions can be 

in column 2 that a "user of the system may reject certain found by ray tracing for which occlusion and reflection 

areas and eliminate erroneous readings caused by multiple artifacts will be present within any single image. For 

reflections from complex geometries". This allows the user demanding high speed applications, the time to acquire and 

access to image for placing "windows". Process multiple viewpoints acquired through rotations and 

In the '772 patent (also using multiple detecting articulation is likely not available, 

elements), the exposure is controUed to balance power on 5^ Other systems utilizing multiple triangulation-based 

detector. detectors are disclosed in the Hata, U.S. Pat. No. 4,731,853, 

Ihe Stern, U.S. Pat. No. 4,643,578, describes a system which uses CCD detectors, and in the article by Kooijman, 
that uses a mask synchronized with the position of a light K., Horijon, J., (PhiUps) "Video Rate Laser Scanner: Con- 
beam and readout location. This method is very effective for siderations on Triangulation, Optics, Detectors, and Process- 
reduction of errors from stray light outside the FOV, and to 55 Circuits", Proc SPIE, VoL 2065, D. Svetkoff ed., Sep- 
mask high background optical noise as might be encoun- tember 1993, p. 253, Sec. 2, Tnangulation OpUcs. 
tered in a welding application. However, the entire position SUMMARY OF THE INVENTION 
sensing area cannot be masked effectively so that severe 

errors can still be present. An object of the present invention is to provide a 

The Penny et al., U.S. Pat. Nos. 4,634,879 and 4,645,917, 60 triangulation-based 3D imaging and processing method and 

describe a means of filtering optical noise similar to Stem, system including filtering and estimation techniques for 3D 

but the system employs photo-multipUer tubes for position imaging so to reduce the measurement errors sensed along 

detection rather than a CCD array. The descanning action the depth position sensitive axis of measurement, hence 

performs the same function as shown by Stem. Only a single substantiaUy improving die overall performance of high 

receiver is used and occlusion effects hmit performance. 65 speed, laser scanned systems. 

The Nakawaga, U.S. Pat. No. 4,553,844, shows a Another object of the present invention is to provide a 

scanning/descanning arrangement for inspecting solder. The triangulation-based 3D imaging and processing method and 
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system which use information about object structure as seen the intensity data from the first and second directions with 

by differences in measured height and gray scale data, and the at least one control algorithm to obtain transformed 

provide a knowledge-based multi -channel signal processing ^leight data. Finally, the method includes the step of trans- 

approach which cither estimates (through weighted average "1^^^"^ ^1'"'^°™' h W ^i^.a'^^'^nJ^jJ^^^^ ™fJr 

of multiple sensor data) or filters each 3D data value for 5 ^^1,:;,^,^^^ S daVr=^^^ at least one trans- 

maxmium contrast depcndmg upon the confidence level of ^^^^^ j^^-gj^j ^^^^ transformed intensity data 

the information. represents at least one transformed gray scale value. 

Yet another object of the present invention is to provide a Still further in carrying out the above objects and other 

triangulation-based 3D imaging and processing method and objects of the present invention, a triangulation-bascd 3D 

system which use a high speed laser scanning system to scan 10 imaging and processing method for generating transformed 

or project a single line of laser light and receive Ught with hei^t and gray scale values regarding an object located at 

exacUy two detectors, yet produce a much higher overall mspection station is provided. The method mcludes the 
speed-Luracy figure of merit than slower mechamcaUy 

scanned point measurement device which use a plurality (2 ^^^^^^ ^ ^^^^^ ^^^^^^ ^^g^^j^^ ^^^^^ 

or more) of detectors or additional detector/transmitter pairs, is ^^^^^^ ^^^^^^ receiving and focusing the first reflected 

hence, maximizing potential utilization. ^^^^ ^^tain a first focused spot of radiant energy at a first 

Yet another object of the present invention is to provide a imaging area and receiving and focusing the second 

triangulation-based 3D imaging and processing method and reflected beam to obtain a second focused spot of radiant 

system which operate in real lime robustly and independent energy at a second imaging area. The method also includes 

of special programming requirements (user intervention and the steps of detecting the position and intensity of the first 

placement of window) which could degrade measurement focused spot of radiant energy at the first imaging area to 

speed, yet allow for sufficient programming and threshold provide a first signal, processing the first signal to generate 

selection so as to optimize filtering and estimation for a first set of height data and a first set of intensity data, 

different imaging and measurement appUcations. detecting the position and intensity of the second focused 

_ , , . *• *■ • # « 25 spot of radiant enercY at the second imaging area to provide 

Yet another object of the present mvenUon is to provide a ^P^^^^ and processing the second^gnal to generate 
tnangulation-based 3D miagmg and processmg method and ^ secondset of height data and a second set of intensity data, 
system which improve the dynamic range by hrnitmg ii^e method further includes the steps of controllably select- 
required use of but not excluding real time hght modulation ^^^^ ^^^^^^j algorithm adapted to process the sets 
devices (direct laser modulation, Acousto-optic, etc.) which ^^^^^ intensity data, processing the first and second 
are used to compress or otherwise limit signals into elec- g^^g of intensity data to obtain a transformed set of intensity 
tronic range, recognizing that overload in one channel need ^1^^^ ^nd processing the first and second sets of height data 
not be an overload in another, and that the largest of two ^nd the first and second sets of intensity data with the at least 
signals may not be the most desirable. one control algorithm to obtain a transformed set of height 

Yet another object of the present invention is to provide a data. Finally, the method includes the step of transmitting 

triangulation-based 3D imaging and processing method and the transformed sets of height and intensity data which arc 

system which, in turn, provides linear and non-linear point- adapted to be processed by an image processing computer, 

wise filtering methods for defect enhancement and texture The transformed set of intensity data represents at least one 

analysis to enhance surface microstructure rather than "aver- transformed gray scale value and the transformed set of 

age our the detail as done with the prior art. height data represents at least one transformed height value. 

Yet stiU another object of the present invention is to ^rther in carrying out the above objects and o^^^ 

. , '^'^ 1 u J oI^ • • ^ ' of the present invention, a system is provided for carrying 

provide a triangulation-based 3D imaging and processing ^^^^^ ^^^^^^ ^^^^^ ^^^^^^^ 

method and system which weight pixels based upon confi- ^^^^^ .^^^^^.^^ ^^^^^^^ 

dence as determmed by a relative match of gray scale and ^^^^ dimensional imaging and measurement using the 

height data, and select transforms based upon the relative 45 p^nciple of triangulation, and specifically relates to the use 

^ of laser scanning systems for high speed imaging, in which 

Yet a further object of the present invention is to provide multiple detection systems are incorporated for improved 

a triangulation-based 3D imaging and processing method accuracy through reduction of the effects of occlusion and 

and system which include a knowledge based 3D sensor spurious reflections. The invention relates to a new and 

which produces a confidence measure which can be used to 50 useful filtering and estimation technique for gray scale and 

determine data validity, or by means of computer control, height data acquired by a laser scanning or structured light 

control of manipulator so as to provide feedback for an sensor where each point in the scerie is viewed by exactly 

articulator if so desired. two sensors positioned on either side of the scan line or 

In carrying out the above objects and other objects of the projected line of light. The method allows for reduction or 

present kivention, a triangulation-based 3D processing 55 elimination of e^ors which otherwise limit the achieva^^^^ 

method for generating trailformed height and gray scale fig»«^ ^^^^ ^f laser scanned systems when compared to 

values is provided, T^e method include! the steps of gen- ^^i.^^^dd^^^^^^^ triangulaUon systems mcoiporaung 

crating a first pair of data values represented by height data ^n. u u- ^ a *u . f^^H.™^ .^a 

and inTensity data obtained from a fi^t direction with respect ^ Th^ ^*Jove objects and other objects features, and advan- 

diiu iiiicuMiy u*ia uuiai i a ' f A i ,,,1,,^. <n t^gcs of thc prcscnt invention are readdy apparent from the 

to an object and generating a second pair of data values 60 following delaUed description of the best mode for carrying 

represented by height data and intensity data obtained frorn invention when taken in connection with the accom- 

a second direction with respect to the object. The method panying drawings, 
also includes the steps of controllably selecting at least one 

control algorithm adapted to process the height and intensity BRIEF DESCRIPTION OF THE DRAWINGS 

data, processing the intensity data from the first and second 65 FIG. 1 is a schematic view of a basic triangulation-based 

directions to obtain transformed intensity data and process- system including a telocentric receiver with a single detec- 

ing the height data from the first and second directions and tor; 
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FIG. 2 is a schematic view of a triangulation-based dual decades of additional range. The preferable method is to 

receiver system which reduces shadow by using at least two "ramp up" the light level and monitor the return with 

receiver and detector subsystems; auxiliary amplifying detectors. The detectors of choice are 

FIG. 3 is a schematic view of a triangulation-based dual avalanche photodiodes, preferably devices with 5 mm or 
receiver system wherein the combined effect of occlusion S largerdiameter suppUed by Advanced Photo nix CorporaUon 

and bulk internal scattering creates asymmetric spot dislri- and having combiner ampUficr/detcctor nse times of 25 NS 

butions at each detector which are not cancelled by averag- 1^^* 

ing sensor channels; As the method of the present invention acquires data by 

FIG. 4 is a schematic view of a triangulation-based dual scanning or projecting a plurality of points (usually 512, but 

receiver system utilizing triangulaUon, wherein the errors ^ ^YPical range of 64 to 1024, but not excludmg longer 

caused by secondary reflection are a complex function of scans or projection), a means of scanmng or projectmg a 

surface geometry, reflectance, and background laser beam is needed. The preferable system previously 

characteristics, including object proximity; described in the Svetkoff et al., U.S. Pat. No. 5,024,529, uses 

FIG. 5 is a schematic view of a triangulation-based dual I ^^^^f'^'^ acousto-optic deflector to produce a high ^eed 

receiver system wherein shiny spherical shaped surfaces ^ying laser spot with a telocentric optical system. Other 

present a problem with large localized reflect^Ince changes ^igh speed scanning mclude polygon laser 

(dynamic range) compounded with background reflectk)D scanners, resonan devices and hologons w^^^^ are mecham- 

^ , j* if * cal defection units are less desirable. Also, projection or 

which may exceed signal from pomt mterest; ^^^^ ^^^^ ^p^^^ cameras for faster 

HG. 6 is a schematic view of a dual sensor tnangulation- 20 ^adout than the typical 15K points per second can be used, 

based imaging system constructed m accordance with the quasi-video rate, full field appUcations. 

present invention and having general numerical and logical ^ _ . ♦ ■41 n 

data filterin ■ system of the present mvention also generally 

^' includes first and second substantially identical sensors or 

RG. 7 is a schematic block diagram of basic curcuitry ^ceivers 106 and 108 which view off-axis. Each of the 
used combine the mulUple sensor signals; 25 receivers 106 and 108 includes an optical delivery and 

FIG. 8 is a schematic block diagram of more specific focusing system and a detector having a position sensitive 

circuitry used to combine the multiple sensor signals and axis for determination of height by position detection and 

having a smaller memory requirement than the circuitry of triangulation principle and for determination of received 

FIG, 7; and laser beam intensity. Optimally, each of the receivers 106 

FIG. 9 is a schematic block diagram of other specific and 108 also includes a high speed ampUfying detector 

circuitry used to combine the multiple sensor signals and (limiter) for laser intensity control, where a small fraction of 

having even a smaller memory requirement that the circuitry the received light is split and delivered to the detector which 

of FIG. 8. preferably includes an avalanche photodiode and its asso- 

ciated pre -amplifier. 

BEST MODE FOR CARRYING OUT THE 35 r , • *u . e xTir- ^ iu o e, 

TNTVPNTTDN particular, m the system of FIG. 6, there is symmetry 

IIN tiiN 1 about the long axis of the scarmed and projected beam, 

Referring now to the drawing figures, there is illustrated resulting in exactly two nearly identical optical arrange- 
in FIG, 6 a triangulation-based imaging and processing mentsfordelivery of light to the detectors each of which has 
system constructed in accordance with the present invention. a position sensing axis. Various optical and electronic posi- 

The system includes a laser transmitter 100 which gen- tion sensitive detectors can be used depending upon speed 

erally includes laser beam generator and electronic modu- and sensitivity requirements, but lateral effect photodiodes 

lation mechanism for laser intensity control, beam deflector are now available which can provide high speed capability 

for scanning the laser beam or projecting a line of laser light, (up to video rates) with good performance provided that the 

and optical focusing means to create a plurality of scanned dynamic range requirement is not extremely large (i.e. very 

points or a projected line of light across an object 102. low level signals). For the two detector system of FEG. 6, 

More specifically, the laser is typically a CXASS IH-B good matching is required along with capability for calibra- 

laser diode such as the Sharp LT015-MD with wavelength tion updates via hardware look-up tables, 

typically in the range of 780 nm-830 nm and with maximum A best implementation of the previously mentioned inten- 
powcr below 50 mw. Advances in semiconductor laser diode 50 sity control method will be determined by a combination of 

technology may allow for shorter wave-lengths (680 nm) the received signals from each of the two detectors, 

which has a primary advantage of alleviating requirements FIG. 5 best illustrates the point where the capability of the 

for enclosures and interlocks. detector 70 is sacrificed. Laser intensity control for enhanc- 

Real time intensity modulation by an autolight control ing the electronic dynamic range is not independent of the 
104 is an option for very wide dynamic range imaging, ss optical dynamic range and the effects of optical crosstalk 

where the directional reflectance varies by more than about resulting from a finite point spread function. 

3 decades. The modulation of the laser diode may be direct. In general, the automatic light control 104 is electronic 

but reduction by more than about one decade below the rated and has inputs derived from the amplifying detectors pre- 

power produces undesirable spatial and temporal modes viously mentioned. The light controll04 produces an output 
which can introduce spatial noise, particularly in diffractive 60 voltage to control the modulator of the transmitter 100 based 

scanning systems, but also in a classical systems using a upon a predetermined function of both channels, (i.e. set 

polygon, galvanometer, or resonant device. light output such that Max(gl,g2)<V maximum and Min 

The preferred means of modulation is with the use of an (gljg2)>V minimum where V maximum and V minimum 

acousto-optic modulator with a beveled structure optimized correspond to upper and lower useful limits of position 
for minimal delay such as 50 ns delay. The control range 65 sensor dynamic range) so that, whenever possible, neither 

offered with such a modulator is much larger than what is channel exceeds a predetermined maximum intensity value 

achievable with direct laser modulation, exceeding 2 set by the maximum useful dynamic range of the position 
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detector, but both channels exceed a lower threshold, Control logic 134 is used to load the desired decision rule 

thereby maintaining a valid data condition. into the look-up table memory, as well as provide quick 

Also, in general, each of the receivers 106 and 108 modifications of the current decision rules based on 

includes signal processing means or circuit which produces calibrations, or to select pre-load decision rules with little 
a digital height value representing the ccntroid of the spot S impact on processing time, 

intensity distribution on the detector, and to produce digital This general implementation provides full functional 

gray scale from integrated distribution or peak intensity encoding of the 3D and gray scale outputs based upon the 

value, 3D and gray scale inputs from both receivers. However, at 

Each position detector require such a signal processing this time, the amount of memory needed may be cost 
means for compressing wide dynamic range signals into a ^° prohibitive for the appUcation it must support. Several 

somewhat limited (typically 10-12 bits at high speed) A-D variations of this implementation exist that can provide the 

converter. A switched gain amplifier system is preferred, desired functionality with minimum memory requirements, 

where each gain (or compression) stage is activated based FIG. 8 shows the preferred embodiment with a much 

upon a comparison with a signal threshold. This allows for smaller memory requirement than the embodiment of FIG. 

good linearity, easy calibration, and good stability with 7. For most applications, modification of the gray scale 

temperature. DC restoration circuitry is used (sometimes signal need not depend upon the calculated 3D values, 

called correlated double sampling) to compensate for offset therefore a look-up table 140 provides a gray scale output 

drift, but more important, to cancel position dependent only dependent upon the two gray scale inputs. The output 

errors arising from charge storage and leakage from high of the look-up table 140 can be the result from cither 

signal values to lower ones. If the correction is applied to receiver only, or a linear or non-linear estimate based upon 

each pixel, substantial reduction in image smear occurs. both channels, such as an average, maximum, or maximum 

Other means of signal processing include log ratio circuits value from each channel, 

which simplify the ratiometric processing to a simple sub- Gray scale- to -Z look-up tables 142 and 144 provide full 

traction operation. gray scale to 3D transformation for each channel respec- 

After digitization of the gray scale and height tively. In addition, this implementation shows two additional 

information, the multiple sensor signals must be combined bits which may be used to encode additional information 

in some way to produce a good estimate of the height value about the gray scale-to -Z relationship, such as gray scale 

at any given point, along with improving gray scale contrast thresholding, which can be passed on to a decision rule 

in the presence of corrupting reflection noise described look-up table 146. For example, if the first receiver 106 has 

earlier wherein it was shown that linear estimation (average a gray scale value less than or greater than a pre-set 

or sum) of values is too simple for complex scenes where threshold, then the decision rule should select the 3D value 

occlusion and reflection problems are present. Furthermore, from the second receiver 108. The number of extra bits 

the averaging (or sums) limit the capability for study of passed on from the gray scale-to-Z look-up tables 142 and 

surface texture when the scanned laser spot size is approxi- 144 are shown as two bits, providing up to 4 tests that can 

mately the size of the surface marks, in some cases it is be applied in the Decision Rule look-up table 146. However, 

desirable to enhance these features, rather than average. any number of bits could be used. 

Hence, it is appropriate to employ conditional decision FIG. 9 shows an alternate embodiment with even less of 

logic in which to select a series of alternative linear or a memory requirement than the embodiment of FIG. 8. The 
non-linear transformations based upon the confidence of the 40 gray scale-to-Z look-up tables 142 and 144 of FIG. 8 are 

sensor data, knowledge of scene reflectance of transmission replaced with comparators ISO and 152, which can provide 

characteristics, or the overall measurement goal. The func- information to a Decision Rule look-up table 154 after 

tion accomplished is both a filter and an estimator as comparing the gray scale data from the first and second 

illustrated in FIG. 6 as a multi-channel filtering and estima- receivers 106 and 108 with known constant values on buses 
tion block 110. A controller 112 controls selected logic or 45 156 and 158. This implementation may be preferred if full 

numeric function via programmed inputs from a host com- gray scale-to-3D transformations are not necessary, 

puter 114. FIG. 7 shows a specific embodiment which A gray scale-to -gray scale look-up table 160 can also 

provides for implementation of a large number of numerical generate an informational bit 162 connected to the Decision 

and combinational logic functions. Rule look-up table 154, such as which of the two receivers 

FIG. 7 illustrates a general implementadon of combining 50 106 and 108 is providing the maximum gray scale value, 

logic for providing filtering and estimation of the 3D (i.e. zl, Although this implementation shows one comparator per 

z2) and gray scale, gSj gs^, information. Hiis implementa- gray scale channel, multiple comparators could be used to 

tion shows the input and output data to be 8 bits of provide multiple threshold data values, including the sped- 

resolution, however this implementation can apply to any fying of a range of gray scale values to select the proper Z 
number of bits for the inputs or outputs. Inputs 120 (zl) and 55 decision rule. The memory requirement of each of the 

122 (gSj) are the 3D and gray scale information from the first implementations of FIGS. 8 and 9 can also be further 

receiver 106, and inputs 124 (z2) and 126 (g^^ are the 3D reduced if the gray scale to gray scale transformation is not 

and gray scale information from the second receiver 108. A needed by replacing the look-up table (140 or 160) with a 

look-up table 128 contains enough memory access for a switch (not shown) to continuously provide the gray scale 
one-to-one correspondence for every input combination to 60 information from either the first receiver 106 or the second 

provide a desired output for the Z data 130 and the gray scale receiver 108. 

data 132. This logic can be used to correct various errors that The logic providing the gray scale and 3D information 

may exist in the system, such as scale and offset errors may have specifically encoded values in the data that may 

between the two sensors, or known offsets in the 3D signify that the gray scale intensity is too high or too low to 
information based upon the gray scale value, as well as 65 provide accurate 3D information, or that the 3D information 

provide the proper decision rules to create the desired output is out of range. In all of the described implementations (i.e. 

for a particular application. of FIGS. 6-9), the transformation tables can pass the special 
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encoded values unchanged to the downstream logic, or even typical using the 90 degree rotation test. The perfomaance is 

make special decisions based upon the appearance of the within typical customer solder paste specifications. 

special values in one or more channels. Spatial filtering such Furthermore, image quality for inspection of microscopic 

as a median filter can be used later to "fillin" the special bumps and BGA technology was improved, allowing copla- 

encoded values based upon the data which can be inferred 5 narity measurements on miniature solder balls having 40 

from the surrounding pixels. ^^^^^^ diameter Reproducibility of 1 micron was found for 

T , u- f r?rr« ^ • a- * \ a « Ihe coplanarity measurement using the above-noted dual 

In general the combiner of FIG. 7 is digital and prepro- ^^^^J^ ^^^^ 5 ^^^^^ ^^^^^^1 ^^^^^^^^^ 

cesses dual channels of 3D and gray scale mformaUon or ^^^^ ^^^^^^ ^^^^ ^ g ^.^^^^ ^^^^1^^^^ 

data to provide estimauon and filtering of ^ay scale and 3D ^^^^^ ^^^^^ improvements arc anticipated with minor 

data prior to computer image analysis. The foUowing typical 10 njodifications of the decision rules. Improved estimates of 

functions are perforated at block 110: ^all diameter were also obtained using the minimum of gray 

a) comparing digitized gray scale values gl and g2 and scale values from the channels for contrast improvement, 
comparing gl and g2 with programmable thresholds Advancements in digital computer architectures and 
indicating selected upper and lower limits of dynamic reduced cost of memory and high speed arithmetic and logic 
range (this range may be the entire sensor dynamic ^jits will result in changes in the implcmcnUtion by sub- 
range or a narrower range of pre-programmed values, stitution of new and faster components, miniaturization, and 
i.e. as narrow as about 3:1 but as wide as 10^-10^; alteration of the sequence in which some operations are 

b) encoding the data point with a special value if both gl performed. Specific modifications well known to those 
and g2 are not within specified limits; skilled in the art include methods for reduction of the 

c) comparing the difference in recorded height values imaging head size, electronic designs allowing for incoipo- 
(zl-z2) with a preprogrammed match tolerance if val- ration of signal processing functions within the imaging 
ues gl and g2 are within the selected lower and upper head, reduction in the memory required within the multi- 
limits of dynamic range; channel processor with alternative use of high speed arith- 

d) selecting at least one logic or numeric function based ^5 ^^^^ ^^^^ storage components for additional 
upon result of comparison of height values zl and z2 images to aUow several functions to be performed on the 
and with a predetermined match tolerance, and com- mput image thus producmg multiple output images, etc, 
parison of gray scale values gl and g2 with prepro- However, the key elements of the invention, which pertain 
grammed thresholds; ^0 ^^'^ selection of logical and numerical operations based 

e) transforming height 'values zl and z2 to a new height 30 ^pon confidence of the 3D and gray scale data and the 
value Z based upon a comparison with a matching threshold knowledge of the underlying physics, result in a significant 
and values of gray scale gl and g2 with predetermined advancement over the prior art, particularly the systems 
thresholds, the transformation to include any or all of the which average the two sensor channels- 
following* While the best mode for carrying out the invention has 

i. weight linear (or afSne) transformations of height 35 been described in detail, those famiUar with the art to which 
values to provide a best esUmate of Z when zl and this invention relates will recognize various alternative 
z2 are within the match tolerance and gray scale designs and embodiments for practicing the invention as 
values gl and g2 are both within a preprogrammed defined by the following claims. 

range, What is claimed is: 

ii. non-linear order statistics (i.e. minimum or 1. A triangulation-based 3-D processing method for gen- 
maximum) when data exceeds a larger tolerance, or erating transformed height data, the method comprismg the 
for contrast improvement if best estimate Z is not steps of: 

required but mismatch is within the tolerance, transmitting a first beam of controlled radiant energy to a 

iii. special codes indicating an invalid data condition surface of an object to obtain first and second reflected 
when the mismatch exceeds a specified tolerance, a 45 beams of radiant energy; 

very important operation, generating a first pair of data values represented by height 

iv. selecting either zl (or z2) if gray scale value g2 (or data and intensity data obtained from the first reflected 
gl) is outside programmed limit of dynamic range beam; 

but other channel is within the limit, generating a second pair of data values represented by 

f) transforming gray scale values gl and g2 to a new gray 50 height data and intensity data obtained from the second 
scale value G based upon a comparison of a matching reflected beam; 

threshold with a specified linear or non-linear controllably selecting at least one control algorithm 

transformation, and to include both weighted linear adapted to process the height and intensity data; and 

transformations of gray values when zl and z2 are processing the height data from the first and second 

within the match tolerance, and non-linear order sta- 55 reflected beams and the intensity data from the first and 

tistics (i.e. minimum or maximum) when data exceeds second reflected beams with the at least one control 

a larger tolerance, or other special codes when the algorithm to obtain transformed height data, 

mismatch exceeds specified tolerance hence indicating 2. The method of claim 1 wherein the transformed height 

an invalid gray scale data condition; data represents a plurality of transformed height values and 

g) storing at least one transformed gray scale value 60 wherein an output image comprises a minimum of the 
G(gl,g2) for every single sample point of the object; transformed height values. 

and 3. The method of claim 1 wherein the transformed height 

h) storing at least one transformed height value Z(zl,z2, data represents a plurality of transformed height values and 
gl,g2) for every single sample point on the object. wherein an output image comprises a maximum of the 

Experimental results with the imaging system based upon 65 transformed height values, 

the conditional decision rules of FIG. 8 have shown reduc- 4. The method of claim 1 wherein the transformed height 

tion in error from an unacceptable 30% to 50% level to 5% data represents a difference in transformed height values, the 
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difference being processed by the at least one control algo- 
rithm to obtain another transformed height value adapted to 
be processed by an image processing computer. 

5. The method of claim 1 further comprising the steps of 
transmitting a second beam of controlled radiant energy to 
the surface of the object to obtain third and fourth reflected 
beams of radiant energy; generating first and second feed- 
back signals based on the third and fourth reflected beams, 
respectively; and utilizing the first and second feedback 
signals to control the intensity of the second beam of 
controlled radiant energy. 

6. The method as claimed in claim 1 further comprising 
the step of: 

processing the intensity data from the first and second 
reflected beams to obtain transformed intensity data, 
wherein the transformed height data represents at least 
one transformed height value and the transformed 
intensity data represents at least one transformed gray 
scale value. 

7. The method as claimed in claim 6 wherein the trans- 
formed height and gray scale values represent a gray scale 
digital image containing the object. 

8. The method as claimed in claim 1 wherein the object 
has a shiny, spherically-shaped surface. 

9. The method as claimed in claim 1 wherein the object is 
a solder pad, 

10. The method as claimed in claim 8 wherein the object 
has a diameter in the range of 10 microns to 0.5 mm. 

11. A triangulation-based 3D processing system for gen- 
erating transformed height data, the system comprising: 

means for transmitting a first beam of controlled radiant 
energy to a surface of an object to obtain first and 
second reflected beams of radiant energy; 

means for generating a first pair of data values represented 
by height data and intensity data obtained from the first 
reflected beam; 

means for generating a second pair of data values repre- 
sented by height data and intensity data obtained from 
the second reflected beam; 

means for controUably selecting at least one control 
algorithm adapted to process the height and intensity 
data; and 
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means for processing the height data from the first and 
second reflected beams and the intensity data from the 
first and second reflected beams with the at least one 
control algorithm to obtain transformed height data. 
5 12. The system of claim 11 wherein the transformed 
height data represents a plurality of transformed height 
values and wherein an output image comprises a minimum 
of the transformed height values. 

13. The system of claim 11 wherein the transformed 
height data represents a plurality of transformed height 
values and wherein an output image comprises a maximum 
of the transformed height values. 

14. The system of claim 11 wherein the transformed 
height data represents a difference in transformed height 
values, the difference being processed by the at least one 

^5 control algorithm to obtain another transformed height value 
adapted to be processed by an image processing computer, 

15. The system of claim 11 fiirther comprising: 
means for transmitting a second beam of controlled 

radiant energy to the surface of the object to obtain 
20 third and fourth reflected beams of radiant energy; first 
and second feedback means for generating first and 
second feedback signals, respectively, based on the 
third and fourth reflected beams, respectively; and 
control means for utilizing the first and second feed- 
25 back signals to control the intensity of the beam of 
controlled radiant energy. 

16. The system of claim 11 further comprising means for 
processing the intensity data from the first and second 
reflected beams to obtain transformed intensity data; 

30 wherein the transformed height data represent at least one 
transformed height value and the transformed intensity data 
represent at least one transformed gray scale value. 

17. The system as claimed in claim 16 wherein the 
transformed height and gray scale values represent a gray 

35 scale digital image containing the object. 

18. The system as claimed in claim 11 wherein the object 
has a shiny spherically-shaped surface. 

19. The system as claimed in claim 11 wherein the object 
is a solder pad. 

40 20. The system as claimed in claim 18 wherein the object 
has a diameter in the range of 10 microns to 0.5 mm. 

* * ♦ * 
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